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Abstract 

The health care field is focusing considerable attention on dietary control, which requires that individuals record what they 
eat. We have developed a novel smartphone application called FoodLog, a multimedia food recording tool that allows users 
to take photos of their meals and to produce textual food records. Unlike conventional smartphone-based food recording 
tools, FoodLog allows users to employ meal photos to help them to input textual descriptions based on image retrieval. In 
this study, we conducted usability experiments to evaluate the routine daily use of FoodLog systems with and without image- 
based assistance. We produced 2 food recording tools: FoodLog with image-based assistance (FL-I) and FoodLog with text 
input only (FL-T). We recruited 18 university students (age = 18-24 years), all of whom performed food recording for the 
first time. The participants used FoodLog on a daily basis for I month. In the subjective evaluation, FL-I had higher average 
scores for questions related to ease of use, fun, frequency of browsing, and intention to continue. In particular, the latter 3 
factors received significantly higher scores with FL-I than with FL-T. In the quantitative evaluation, the daily average number 
of meal events and food records did not differ significantly between FL-I and FL-T. A detailed analysis of the individual records 
showed that I participant produced 3 times as many records using FL-I compared with FL-T. The subjective assessment 
showed that our new tool, which fully exploits the use of images, is a promising method for food recording. 
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Obesity is a growing global concern. Thus, dietary control is 
receiving considerable attention in the health care field. It is 
considered that recording our food intake is vital for control- 
ling eating and health management. The traditional methods 
used for food recording rely on human memory, where sub- 
jects complete forms or answer questions in an interview to 
remember what they consumed. Manually recording detailed 
information about all meals is a tedious task, and it is diffi- 
cult for people to adhere to this process over a long period. 
Thus, there is a strong demand for food intake recording 
using information technology.^ 

The rapid spread of mobile devices such as smartphones, 
which individuals may use to record personal information, 
means that they could be used as a data collection platform.^'^ 
A previous study showed that adolescents prefer dietary 
assessments using technology, such as PDAs, rather than tra- 
ditional paper-based food recording."^ At present, quite a few 
smartphone apps are available for dietary self-manage- 
ment.^'^ However, the recording of food items is normally 



text based in current food recording tools. From a usability 
perspective, the burden of text-based data input is too high to 
promote continuous use. Thus, automatic or semiautomatic 
tools are being investigated. '^"^^ To the best of our knowledge, 
however, none of these tools has been made publicly 
available. 

To assist food recoding, we developed a new smartphone- 
based appHcation called FoodLog. Our system allows a user 
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Figure I . Screenshots from FoodLog. Left to right: (A) calendar view; (B) view of a single day, where the food images are shown with 
descriptions of the food names, the portion sizes, and their calorific contents; and (C) a food image with labels showing the calorific 
contents of food items. (The food names are shown in Japanese.) 



to take a photo of a meal and to interact with the photo on the 
smartphone screen, so they can input the names of food items 
with the assistance of image retrieval. Thus, food records can 
be viewed as a photographic food diary. (At present, our sys- 
tem is publicly available in Japan.) Compared with existing 
smartphone-based food recording applications, our system is 
very different because it fully exploits food pictures. In this 
study, we performed a comparative analysis of the usability 
of this new smartphone-based application for food 
recording. 

Methods 

Recruitment and Study Design 

A sample of healthy university students aged 1 8 to 24 years 
was recruited from various schools at the University of 
Tokyo (11 males, 7 females) after we requested participants. 
Eighteen students applied to participate in the experiments, 
and no selection procedures were applied. The majority of 
the participants came from the School of Liberal Arts and 
Science. All of the participants were smartphone users. None 
of the participants had used FoodLog before and they had no 
experiences of food recording. On the first day, the partici- 
pants came to our laboratory and either FoodLog with 



image-based assistance (FL-I) or FoodLog with text-based 
input only (FL-T) (the "apps") was installed on their per- 
sonal smartphones according to our instructions. They 
used the apps in their daily life. After approximately 2 
weeks, the participants returned to the laboratory and they 
removed their app, before installing the other app, which 
they used for approximately 2 more weeks. Half of the par- 
ticipants started with FL-I and the other half started with 
FL-T. At the end of the trials, the participants returned to 
our laboratory and completed questionnaires. The food 
intake records and photos were uploaded automatically to 
a server, and a statistical analysis of the food records was 
conducted. The participants received no reminders or other 
interventions during the experiments. The study methods 
were approved by the University of Tokyo Institutional 
Review Board, and informed consent was obtained from 
each participant. 

FoodLog: How It Works 

FoodLog uses photos of meals in 2 ways: as a visual food 
diary and to help the user to input the names of food items. 
Figure 1 shows examples of FoodLog screenshots. The cal- 
endar view shows that the visual food diary is easy to under- 
stand. Descriptions of food items in each image are shown in 
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Figure 2. Flowchart of food recording with FoodLog. 



Figure IB. In addition, Figure IC demonstrates that the calo- 
rific contents of each food item are shown. 

FoodLog has 2 modes for inputting the names of food 
items: a text-based mode and an image-assisted mode. Both 
modes are shown in Figure 2, where the text-based mode is 
shown below the dashed line and the image-assisted mode is 
above that line. Figure 2 shows that the text-based mode is 
the baseline method, while the image-assisted mode helps 
via image retrieval. 

The text-based mode is almost the same as existing tools. 
In the text-based mode, the food name and portion size are 
required inputs. Part of the name is sufficient as an input 
because partial match searching is enabled. A standard data- 
base is searched when part of the name of the dish is entered 
as text. A personal database based on the user's history is 
also searched simultaneously. Both of the search results are 
displayed as a list. The user can select from the list and select 
the portion size. Free text input is also available if no items 
are found in the databases. 

We developed the image-assisted mode to make these 
interactions simpler and more intuitive using an image 
retrieval technique. The system is operated as follows. (1) 
Take a photo of a meal. (2) Specify the food domain by 
touching the screen and adjust the size of the domain (Figure 
3 A). A visual similarity search in the personal image data- 
base is performed by the smartphone, and the top 20 results 
are shown as a list (Figure 3B). (3) Choose the appropriate 
food from the list and adjust its portion size. The specified 
food domain is then registered in the personal image data- 
base for the next search. The visual search process was suf- 
ficiently fast because the time required to search was much 
less than 1 second. 

Provided that the food can be identified in the candidate 
list, the user operations are simply touch, adjust, and select. 
However, if the target food is completely new or the visual 
search results do not contain the appropriate food, the user 
employs the text-based mode to specify the food domain. 



Experimental Comparison of Usability 

FoodLog is a unique system that fully exploits food photos, 
and it also has a text-based input mode. As shown in Figure 
2, the text-based mode, which has almost the same function- 
ality as conventional tools, is overlaid with the image- 
assisted mode. Without the image-assisted mode, FoodLog is 
simply a text-based food recording tool. We produced 2 tools 
to investigate the usability of our system: FL-I and FL-T. 

Data Used in the Evaluations 

We performed quantitative and subjective usability evalu- 
ations. In the quantitative evaluation, we compared the 
number of food items and meal events recorded using the 
2 tools. The food records made by the subjects were 
uploaded to our server automatically. We investigated 
whether there was a difference in the amount of FL-I and 
FL-T usage either on the average of all the subjects or 
individually. 

The subjective evaluation was based on a questionnaire, 
which contained the following questions: (1) "How easy was 
it to make food records?" (2) "How much fun did you have 
making food records?" (3) "How often did you browse your 
food records?" and (4) "Do you want to continue food 
recording?" The subjects answered each question using a 
5 -point scale: positive - 5, weakly positive = 4, neutral = 3, 
weakly negative = 2, and negative = 1. 

The 1 8 students who participated in these experiments 
had no previous experiences of food recording or the 
FoodLog system. During the approximately 1 -month 
experiments, we did not send any reminders to encourage 
food recording. At the end of the trial, 6 of the participants 
had insufficient records, that is, the average number of 
food item records or images was less than 1 .5 per day. We 
excluded these 6 participants from the statistical 
analysis. 
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Figure 3. Left to right: (A) the specified food domain; (B) results of the visual search. As shown in A, the food domain is specified by 
touching the screen, where the area outside the specified domain is darkened. In this example, the target is in the second rank of the list 
shown in B. 



Results 

Subjective Evaluation 

The results of the subjective evaluations of FL-I and FL-T 
are shown in Figure 4. The sample size was small, but the 
average evaluations for FL-I were better than those for 
FL-T. The responses to the following question categories 
differed significantly: A2, the amount of fun; A3, fre- 
quency of browsing; and A4, intention to continue use. For 
question A2, all of the subjects gave scores of 4 or 5 to 
FL-I, and the difference between FL-I and FL-T was par- 
ticularly noticeable. Although the difference was not sta- 
tistically significant, FL-I had a better average response 
than FL-T for question Al, that is, ease of use. The differ- 
ences in the response scores given by individuals, that is, 
the score for FL-I minus that for FL-T, are shown in 
Figure 5. 

Quantitative Evaluation 

The results of the quantitative evaluation are shown in Figure 
6. The average number of meals recorded per day was 2.7 
with both FL-I and FL-T. The average numbers of food 



Average Score 




Al A2 A3 A4 



Figure 4. Subjective evaluation results, showing the average 
scores for ease of use (A I), fun (A2), frequency of browsing (A3), 
and intention to continue use (A4). The error bars represent the 
95% confidence intervals. 

records were 6.8 and 6.3 for FL-I and FL-T, respectively. The 
number of images taken per day was 3.0 with FL-I. 
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Figure 5. Histogram showing the individual differences in the 
scores for question Al, that is, the score for FL-I minus that 
for FL-T. The positive scores indicate that FL-I received better 
evaluations than FL-T. 
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Figure 7. Histogram showing ratios of the average number of 
records (B2) made by individuals using FL-I relative to those made 
using FL-T. Ratios > 1.0 indicate that more food records were 
captured using FL-I than FL-T. 




Figure 6. Average numbers of meal events (Bl), food records 
(B2), and images (B3) per day. The error bars represent the 95% 
confidence intervals. 



Figure 7 shows the histogram of the ratios of the average 
number of records made by individuals using FL-I relative to 
those made using FL-T. 

Discussion 

To the best of our knowledge, FoodLog is the first publicly 
available smartphone app that allows image retrieval to help 
the user to input food records. The food diary display also 
makes full use of photos, which is visually appealing. This is 
the first study to compare the usability of image-assisted 
food recording with standard text-based food recording. 

The subjective evaluation detected significant differences 
in the responses to questions A2, A3, and A4 for FL-I and 
FL-T. The remarkable difference between FL-I and FL-T for 
question A2 indicated that all of the participants had fun 
when using FL-I. The responses to question A3 showed that 



the participants browsed their food records more frequently 
with FL-I than FL-T. The responses to question A4 showed 
that the participants were more positive about their intention 
to continue using FL-I than FL-T. These results are very 
encouraging for the further improvement of FL-I. In their 
comments, 6 of the participants wrote that they had fun when 
using the tool and they frequently browsed the food photos in 
their records. 

The difference in the responses to question Al was not 
statistically significant, but FL-I received a better average 
score than FL-T, and the analysis of the individual scores 
presented in Figure 5 shows that more participants tended to 
consider that FL-I was easier to use than FL-T. The com- 
ments made by the participants who rated FL-T more highly 
than FL-I indicated that they sometimes experienced dis- 
comfort if they had to take photos of meals. Thus, using the 
system for 2 weeks might not have been sufficient to become 
accustomed to its use. 

The daily activities of the participants were assessed in 
the quantitative evaluation. FL-I and FL-T were used at the 
same frequency based on the average number of daily meal 
events recorded. In addition, the average number of food 
records did not differ between the 2 methods. However, the 
distribution of the individual variations shown in Figure 7 
indicates that FL-I users tended to generate more food 
records, and 1 of the participants produced 3 times as many 
records using FL-I compared with FL-T. Thus, FL-I appeared 
to be effective in encouraging the generation of food records. 

Finally, the comments in the questionnaires indicated that 
5 participants considered that using the system made them 
more aware of what they eat and the calorific contents of 
foods, although they had no previous experiences of food 
recording before the experiments. Six participants suggested 
that the interaction with the FL-I interface could be improved, 
and 4 indicated that the variety of food items in the common 
standard dataset should be increased. 
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Conclusion 

Self-monitoring and dietary control play important roles in the 
prevention of obesity. The use of smartphones has spread rap- 
idly throughout society and they have become highly personal 
devices for many people. Self-monitoring tools based on 
smartphones are available to monitor various daily activities, 
such as exercise, food, and sleep, as well as vital signs data. 
Food recording is a time-consuming and challenging task even 
when such tools are used. In this study, we evaluated our new 
food recording tool, FoodLog, which helps users to input data 
based on image retrieval. We evaluated the diet-related behav- 
iors of participants for 1 month using food recording tools 
with and without image assistance, which were compared 
based on a subjective assessment and a quantitative evalua- 
tion. The subjective assessment detected significant differ- 
ences between the 2 methods. We obtained promising results 
for the new tool with image-based assistance during regular 
daily use. Further improvements of this tool may facihtate bet- 
ter dietary self-management and monitoring by a third party, 
which could be beneficial for the general population. 

In this study, the subjects were healthy university students 
because our evaluation was simply a usability study, which 
assessed the food recording tool with/without image assis- 
tance. The tool could be beneficial in situations that require the 
recording of food intake. FoodLog can be used independently 
or it may be integrated into a system with other functions, such 
as the management of exercise and vital signs. It can also be 
targeted at subjects with obesity or diabetes to facilitate self- 
management or monitoring by a third party. A separate study is 
evaluating a system for diabetics that integrated FoodLog. 
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FL-I, FoodLog with image-based assistance; FL-T, FoodLog with 
text-based input only; PDA, personal digital assistant. 
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